Techniques for improved lighting models for appearance capture

ABSTRACT

Various embodiments include a system for rendering an object, such as human skin or a human head, from captured appearance data. The system includes a processor executing a near field lighting reconstruction module. The system determines at least one of a three-dimensional (3D) position or a 3D orientation of a lighting unit based on a plurality of captured images of a mirror sphere. For each point light source in a plurality of point light sources included in the lighting unit, the system determines an intensity associated with the point light source. The system determines captures appearance data of the object, where the object is illuminated by the lighting unit. The system renders an image of the object based on the appearance data and the intensities associated with each point light source in the plurality of point light sources.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority benefit of the United StatesProvisional Patent Application titled, “IMPROVED LIGHTING MODELS FORAPPEARANCE CAPTURE,” filed on Dec. 16, 2021 and having Ser. No.63/290,643. The subject matter of this related application is herebyincorporated herein by reference.

BACKGROUND Field of the Various Embodiments

Various embodiments relate generally to appearance capture systems and,more specifically, to techniques for improved lighting models forappearance capture.

Description of the Related Art

One aspect of rendering realistic human faces is to capture theappearance of the skin, which has a complex multi-layer structure withcombined surface and subsurface reflectance properties. Appearancecapture techniques estimate geometry and reflectance properties ofvarious objects (such as a human head, facial skin, and/or the like) byperforming a computationally intensive inverse rendering optimization inwhich one or more images are re-rendered a large number of times andcompared to real images captured by multiple cameras at variouspositions and orientations. While state-of-the-art appearance capturetechniques have achieved impressive results, such techniques often makesimplifying assumptions that do not always hold in reality, even thoughthese simplifying assumptions help to reduce computational complexityand rendering time.

One such simplifying assumption is that light sources are often modeledas a collection of point light sources (referred to as an environmentmap) that are distant from the face. The light rays from each of theselight sources are, therefore, assumed to strike all points on the faceat the same incident angle (direction). Further, the directionaldistribution of incoming light is assumed to be spatially invariantwithin a small capture volume and, thus, easier to calibrate by takingdirectional samples from a single three-dimensional (3D) location. Bycontrast, appearance capture studios often have light sources that arerelatively near to the object being captured. Therefore, the incidentlighting angles from a single source indeed vary across the face.Assuming that the light rays from each light source are, instead,parallel with one another leads to rendering errors.

Another simplifying assumption presents when polarization is used fordiffuse-specular reflectance separation in appearance capture systems.When an object being captured is illuminated with polarized light, andcaptured with a camera fitted with a cross-polarized lens, much of thespecular reflectance of the lighting is filtered out, and the diffusereflectance of the lighting remains. This separation of specularreflectance and diffuse reflectance is often assumed to be perfect, suchthat a diffuse albedo can be directly fitted to the cross-polarizedimagery captured by the cameras. The specular reflectance is estimatedonce the diffuse reflectance is known. However, under linearly polarizedillumination, this separation is view-dependent because the optimalpolarizer orientation for each light source is different for each cameraview. Therefore, the perfect separation assumption does not hold ingeneral multi-view setups, where some specular reflectance remains inthe cross-polarized imagery. In some examples, the amount of specularreflectance that remains in the cross-polarized imagery can besignificant, particularly for diffuse-specular reflectance separation atcamera angles that view the face of the object from below.

Yet another simplifying assumption is that appearance capture techniquesoften only account for lighting coming directly from the light sources.Indirect lighting, where light strikes one portion of the face, reflectsoff the face, and then illuminates a different portion of the face isoften ignored. Similarly, the effect of multiple subsurface bounces oflight is also often ignored. As a result, some real-world lightingeffects such as soft shadows and color bleeding, are not accuratelymodeled, resulting in overshoot in recovered diffuse albedo andincomplete specular intensity. Such inaccuracies lead to furtherrendering errors.

As the foregoing illustrates, what is needed in the art are moreeffective techniques for capturing appearance data for the purpose ofrealistic rendering of human faces.

SUMMARY

Various embodiments of the present disclosure set forth acomputer-implemented method for rendering an object, such as human skinor a human head, from captured appearance data. The method includesdetermining at least one of a three-dimensional (3D) position or a 3Dorientation of a lighting unit based on a plurality of captured imagesof a mirror sphere. The method further includes, for each point lightsource in a plurality of point light sources included in the lightingunit, determining an intensity associated with the point light source.The method further includes capturing appearance data of the object,where the object is illuminated by the lighting unit. The method furtherincludes rendering an image of the object based on the appearance dataand the positions and intensities associated with each point lightsource in the plurality of point light sources.

Various embodiments of the present disclosure set forth acomputer-implemented method for rendering an object, such as human skinor a human head from captured appearance data comprising a plurality oftexels. The method includes determining texture coordinates of a firstvector originating from a first texel where the first vector intersectsa second texel. The method further includes rendering the second texelfrom a viewpoint of the first texel based on appearance data at thesecond texel. The method further includes based on the rendering of thesecond texel, determining an indirect lighting intensity incident to thefirst texel from the second texel. The method further includes updatingappearance data at the first texel based on a direct lighting intensityand the indirect lighting intensity incident to the first texel from thesecond texel. The method further includes rendering the first texelbased on the updated appearance data at the first texel.

Other embodiments include, without limitation, a system that implementsone or more aspects of the disclosed techniques, and one or morecomputer readable media including instructions for performing one ormore aspects of the disclosed techniques, as well as a method forperforming one or more aspects of the disclosed techniques.

At least one technical advantage of the disclosed techniques relative tothe prior art is that, with the disclosed near field lightreconstruction, lighting is more accurately modeled for multiple nearfield lighting unit. As a result, a field of near positional lightingunits is reconstructed that more faithfully represents the true lightsin the scene as compared to a conventional distant environment map. Thedisclosed polarization modeling more accurately models specularreflectance present in polarized lighting units, thereby improvingappearance and reducing rendering error relative to techniques thatmodel only the diffuse reflectance from polarized lighting units. Thedisclosed texture space indirect illumination more accurately modelsfacial characteristics where light reflects off one portion of a facialmodel and strikes another portion of the facial model. Another technicaladvantage relative to the prior art is that the disclosed techniques,whether employed individually or in combination, provide improvedaccuracy and realism when rendering objects from appearance capturedata, such as human faces, body parts, and/or other objects, with onlymodest increases in computational cost relative to prior techniques. Thedisclosed techniques account for the impact on inverse rendering qualitywhen departing from idealized conditions, leading to more realisticmodels that more accurately capture and render facial appearance. Theseadvantages represent one or more technological improvements over priorart approaches.

BRIEF DESCRIPTION OF THE DRAWINGS

So that the manner in which the above recited features of the variousembodiments can be understood in detail, a more particular descriptionof the inventive concepts, briefly summarized above, may be had byreference to various embodiments, some of which are illustrated in theappended drawings. It is to be noted, however, that the appendeddrawings illustrate only typical embodiments of the inventive conceptsand are therefore not to be considered limiting of scope in any way, andthat there are other equally effective embodiments.

FIG. 1 is a block diagram of a computer system configured to implementone or more aspects of the various embodiments;

FIG. 2 is a more detailed view of the system memory of FIG. 1 ,according to various embodiments;

FIG. 3 is a schematic diagram of an appearance capture studio, accordingto various embodiments;

FIGS. 4A-4B illustrates techniques for reconstructing a near fieldlighting unit, according to various embodiments;

FIG. 5 illustrates techniques for modeling specular components of apolarized lighting unit, according to various embodiments;

FIG. 6 illustrates techniques for modeling indirect lighting of anobject being captured, according to various embodiments;

FIG. 7 is a flow diagram of method steps for reconstructing near fieldlighting, according to various embodiments;

FIG. 8 is a flow diagram of method steps for modeling specularreflection of polarized light, according to various embodiments; and

FIG. 9 is a flow diagram of method steps for determining indirectlighting effects during appearance capture, according to variousembodiments.

DETAILED DESCRIPTION

The following description contains specific information pertaining toimplementations in the present disclosure. One skilled in the art willrecognize that the present disclosure may be implemented in a mannerdifferent from that specifically discussed herein. The drawings in thepresent application and their accompanying detailed description aredirected to merely exemplary implementations. Unless noted otherwise,like or corresponding elements among the figures may be indicated bylike or corresponding reference numerals. Moreover, the drawings andillustrations in the present application are generally not to scale, andare not intended to correspond to actual relative dimensions.

In the following description, numerous specific details are set forth toprovide a more thorough understanding of the various embodiments.However, it will be apparent to one skilled in the art that theinventive concepts may be practiced without one or more of thesespecific details.

System Overview

FIG. 1 illustrates a computer system 100 configured to implement one ormore aspects of the various embodiments. As shown, the computer system100 includes, without limitation, a central processing unit (CPU) 102and a system memory 104 coupled to a parallel processing subsystem 112via a memory bridge 105 and a communication path 113. The memory bridge105 is further coupled to an I/O (input/output) bridge 107 via acommunication path 106, and the I/O bridge 107 is, in turn, coupled to aswitch 116.

In operation, the I/O bridge 107 is configured to receive user inputinformation from one or more input devices 108, such as a keyboard, amouse, a touchpad, a microphone, a joystick, etc., and forward the inputinformation to the CPU 102 for processing via the communication path 106and the memory bridge 105. The switch 116 is configured to provideconnections between the I/O bridge 107 and other components of thecomputer system 100, such as a network adapter 118 and various add-incards 120 and 121. Although two add-in cards 120 and 121 areillustrated, in some embodiments, the computer system 100 may onlyinclude a single add-in card or more than two add-in cards.

As also shown, the I/O bridge 107 is coupled to a system disk 114 thatmay be configured to store content, applications, and data for use byCPU 102 and parallel processing subsystem 112. As a general matter, thesystem disk 114 provides nonvolatile storage for applications and dataand may include fixed or removable hard disk drives, flash memorydevices, and CD-ROM (compact disc read-only-memory), DVD-ROM (digitalversatile disc-ROM), Blu-ray, HD-DVD (high-definition DVD), or othermagnetic, optical, or solid state storage devices. Finally, although notexplicitly shown, other components, such as universal serial bus orother port connections, compact disc drives, digital versatile discdrives, movie recording devices, and the like, may be connected to theI/O bridge 107 as well.

In various embodiments, the memory bridge 105 may be a Northbridge chip,and the I/O bridge 107 may be a Southbridge chip. In addition,communication paths 106 and 113, as well as other communication pathswithin the computer system 100, may be implemented using any technicallysuitable protocols, including, without limitation, AGP (AcceleratedGraphics Port), HyperTransport, or any other bus or point-to-pointcommunication protocol known in the art.

In some embodiments, the parallel processing subsystem 112 comprises agraphics subsystem that delivers pixels to a display device 110 that maybe any conventional cathode ray tube, liquid crystal display,light-emitting diode display, or the like. In such embodiments, theparallel processing subsystem 112 incorporates circuitry optimized forgraphics and video processing, including, for example, video outputcircuitry. Such circuitry may be incorporated across one or moreparallel processing units (PPUs) included within the parallel processingsubsystem 112. In other embodiments, the parallel processing subsystem112 incorporates circuitry optimized for general purpose and/or computeprocessing. Again, such circuitry may be incorporated across one or morePPUs included within the parallel processing subsystem 112 that areconfigured to perform such general purpose and/or compute operations. Inyet other embodiments, the one or more PPUs included within the parallelprocessing subsystem 112 may be configured to perform graphicsprocessing, general purpose processing, and compute processingoperations. The system memory 104 may include at least one device driverconfigured to manage the processing operations of the one or more PPUswithin the parallel processing subsystem 112.

In various embodiments, the parallel processing subsystem 112 may be orinclude a graphics processing unit (GPU). In some embodiments, theparallel processing subsystem 112 may be integrated with one or more ofthe other elements of

FIG. 1 to form a single system. For example, the parallel processingsubsystem 112 may be integrated with the CPU 102 and other connectioncircuitry on a single chip to form a system on chip (SoC).

It will be appreciated that the system shown herein is illustrative andthat variations and modifications are possible. The connection topology,including the number and arrangement of bridges, the number of CPUs, andthe number of parallel processing subsystems, may be modified asdesired. For example, in some embodiments, the system memory 104 couldbe connected to the CPU 102 directly rather than through the memorybridge 105, and other devices would communicate with the system memory104 via the memory bridge 105 and the CPU 102. In other alternativetopologies, the parallel processing subsystem 112 may be connected tothe I/O bridge 107 or directly to the CPU 102, rather than to the memorybridge 105. In still other embodiments, the I/O bridge 107 and thememory bridge 105 may be integrated into a single chip instead ofexisting as one or more discrete devices. In some embodiments, anycombination of the CPU 102, the parallel processing subsystem 112, andthe system memory 104 may be replaced with any type of virtual computingsystem, distributed computing system, or cloud computing environment,such as a public cloud, a private cloud, or a hybrid cloud. Lastly, incertain embodiments, one or more components shown in FIG. 1 may not bepresent. For example, the switch 116 could be eliminated, and thenetwork adapter 118 and add-in cards 120, 121 would connect directly tothe I/O bridge 107. The operating system 140 may be, e.g., Linux®,Microsoft Windows®, or macOS®.

Improved Lighting Models for Appearance Capture

Various embodiments are directed to techniques for capturing facialappearance data for the purpose of realistic rendering of human faces.The embodiments include improved lighting models for appearance capturethat more accurately represent the lighting, while minimally increasingcomputational cost. The disclosed techniques can be employed in anycombination.

In a first technique, referred to as “near field light reconstruction,”a computer system uses multi-view images and triangulation to preciselymeasure the 3D locations of lighting unit for high-fidelity appearancecapture. With this technique, the geometry of each lighting unit isrepresented as a rectangular 3D surface with known width and height(from manual measurements) and with position and orientation informationin 3D space that is determined from multi-view images of a mirrorsphere.

In a second technique, referred to as “polarization modeling,” acomputer system models a specular component in cross-polarized viewsrather than assuming purely diffuse reflectance for cross-polarizedviews. In a cross-polarized view, the lighting unit is polarized in onedirection and the cameras are fitted with filters that are polarized ina different cross-polarized direction. Although most reflectance in across-polarized view is diffuse reflectance, the second technique modelsthe remaining specular reflectance.

In a third technique, referred to as “texture space indirectillumination,” a computer system models indirect lighting that resultsfrom light reflecting off one portion of the face and illuminating adifferent part of the face, such as when light reflects off the nose andindirectly illuminates the inner cheek. This technique can beincorporated into different inverse-rendering pipelines.

FIG. 2 is a more detailed view of the system memory 104 of FIG. 1 ,according to various embodiments. As shown, the system memory 104includes, without limitation, an appearance capture application 210,which further includes a near field lighting reconstruction module 220,a polarization modeling module 222, and a texture space indirectillumination module 224. The appearance capture application 210, nearfield lighting reconstruction module 220, polarization modeling module222, and texture space indirect illumination module 224, when executedby CPU 102, perform one or more operations associated with thetechniques described herein. In some examples, the appearance captureapplication 210 is configured to capture facial appearance data for ahuman head. When performing the operations associated with the disclosedtechniques, the appearance capture application 210, near field lightingreconstruction module 220, polarization modeling module 222, and texturespace indirect illumination module 224 store data in and retrieve datafrom system memory 104 and/or system disk 114.

In operation, appearance capture application 210 captures appearancedata for an object such as a human face. Appearance capture application210 captures this appearance data from an appearance capture studio thatlights the object with unpolarized, passive illumination and/orpolarized, passive illumination. In some examples, the appearancecapture studio includes a number of lighting units that are placed at apredetermined distance from the object being captured. For example, theappearance capture studio could include 32 bar lights that are arrangedas 16 pairs of bar lights. The 16 pairs of bar lights could be arrangedat various positions and orientations at various distances from theobject. Each bar light is fitted with a horizontal, linear polarizationfilter. In some examples, the appearance capture studio further includesa number of image capture devices, such as still cameras, video cameras,and/or the like, to capture images of the object as the object isilluminated by the lighting units. For example, the appearance capturestudio could include twelve video cameras that are arranged into 4triplets, where each triplet of cameras includes: (i) a narrow baselinestereo camera pair that is not polarized; and (ii) a central camera, asshown and discussed in conjunction with FIG. 3 . The stereo camera paircaptures full facial reflectance and is also used for stereoreconstruction. The central camera is cross-polarized with respect tothe polarization filter on the lighting units. The central camera mainlycaptures diffuse reflection. In some examples, the number and types oflighting units and image capture devices may vary, with the number ofimage capture devices affecting the number of triplet arrangements.

In an optional process, appearance capture application 210 models theillumination of the lighting units as a collection of distantdirectional sources in a typical latitude-longitude environment map.Appearance capture application 210 computes this environment map using afrontal unpolarized image capture to capture a high-dynamic range (HDR)image of a mirror sphere with known radius, placed at a single locationwhere the object, such as a human face, is subsequently placed forappearance capture. By using multiple calibrated image capture devices,appearance capture application 210 triangulates the 3D position of themirror sphere. The appearance capture application 210 projects 3D rays,also referred to herein as vectors, through the pixels of the HDR imagecapture devices. The 3D rays reflect off the surface of the mirrorsphere and, based on the reflections, appearance capture application 210determines the incoming light directions associated with the pixelintensities of the HDR image capture devices.

Appearance capture application 210 measures the illumination of thelighting units using an approach of capturing multiple exposures of amirror sphere from a particular camera view. In some examples, in orderto improve the speed of appearance capture, appearance captureapplication 210 can compute a sparse representation of the lightingunits, such as by computing the lighting units as a limited resolutionof a fixed number of lighting directions uniformly distributed over thefrontal hemisphere of the object being captured.

In order to improve the appearance parameters, appearance captureapplication 210 implements a custom renderer that operates in UVtexture-space, and reconstructs a diffuse albedo, specular albedo(intensity), and high-resolution normal map. Further, as describedherein, appearance capture application 210 improves rendering quality byexecuting any one or more of near field lighting reconstruction module220 to reconstruct the near field lighting units, also referred toherein as near field lighting units, polarization modeling module 222 tomore accurately model the light polarization, and texture space indirectillumination module 224 to account for indirect illumination of theobject being captured.

In operation, near field lighting reconstruction module 220 improves thefirst approximation lighting model of a distant light source environmentmap by explicitly reconstructing the 3D geometry and intensities of thenear field lighting units. Near field lighting reconstruction module 220captures multi-view images and employs triangulation to measure the 3Dlocations of the lighting units for high-fidelity appearance capture.For the example and non-limiting appearance capture studio describedherein, near field lighting reconstruction module 220 represents thegeometry of each lighting unit as a light-emitting diode (LED) bar lightwith a rectangular 3D surface of known width and height. The width andheight of the lighting unit can be determined a priori from manualmeasurements, specification data sheets, and/or the like. Near fieldlighting reconstruction module 220 determines the position andorientation of each lighting unit in 3D space based on multi-view imagesof the mirror sphere. In addition, near field lighting reconstructionmodule 220 models the near field lighting unit as a fixed number ofpoint light sources that are uniformly placed inside each lighting unit.Near field lighting reconstruction module 220 calibrates the intensitiesof these point light sources using captured HDR images. In someexamples, the intensities are further fine-tuned during appearancecapture, in order to better match the rendered images to the realimages, as described herein.

At a first step, near field lighting reconstruction module 220determines the position and orientation of each lighting unit. To locateeach lighting unit in 3D space, near field lighting reconstructionmodule 220 identifies and annotates the corner vertices of each lightingunit as the corner vertices that appear on the multi-view images of themirror sphere, as captured by the image capture devices. For eachidentified and annotated corner vertex, near field lightingreconstruction module 220 projects 3D rays from each image capturedevice through the identified and annotated corner vertex and reflectsthe 3D rays off of the mirror sphere. Near field lighting reconstructionmodule 220 sets the position of the current corner vertex as theposition in 3D space that is closest to all reflected rays. In someexamples, near field lighting reconstruction module 220 determines theclosest position by regressing the reflected rays to a 3D position ofthe lighting unit by applying least squares regression to the reflected3D rays. Near field lighting reconstruction module 220 repeats thisprocess for all remaining corner vertices for each lighting unit. Inthis manner, near field lighting reconstruction module 220 optimizes forthe 3D light source positions that minimize the corner 3D rayintersection distances for the corresponding corner vertices of eachlighting unit. In addition, near field lighting reconstruction module220 determines the orientation of each lighting unit. Based on the 3Dpositions of the corner vertices of the lighting unit, near fieldlighting reconstruction module 220 determines the plane in 3D space thatintersects the corner vertices. Based on this plane, near field lightingreconstruction module 220 computes a normal vector of the plane. Nearfield lighting reconstruction module 220 selects the direction of thenormal vector that is towards the center of the mirror sphere. In someexamples, near field lighting reconstruction module 220 determines theorientation of each lighting unit by weakly enforcing a regularizationconstraint that orients the normal vector of the plane of the lightingunit towards the center of the mirror sphere, while accounting forpotential inaccuracies in the annotations of the corner vertices.

In this regard, near field lighting reconstruction module 220 initiallydetermines the 3D position of the mirror sphere from multi-viewannotations which may be inaccurate. Therefore, near field lightingreconstruction module 220 jointly optimizes for the position of themirror sphere during light reconstruction of the near field lightingunits. Upon completion of jointly optimizing for the position andorientation of the lighting units and the mirror sphere, near fieldlighting reconstruction module 220 generates a set of 3D orientedshapes, where each shape represents the geometry of the correspondinglighting unit.

At a second step, near field lighting reconstruction module 220 samplesa set of point light sources inside each lighting unit. In someexamples, near field lighting reconstruction module 220 samples tenpoint light sources inside each lighting unit. Alternatively, the numberof point light sources per lighting unit can be more than ten or lessthan ten. Near field lighting reconstruction module 220 captures an HDRimage of the mirror sphere with one of the frontal unpolarized imagecapture devices. For each pixel in this HDR image, near field lightingreconstruction module 220 projects a 3D ray towards the mirror sphereand assigns the observed pixel intensity to the point light source thatis nearest to the reflected ray. Each point light source can accumulatethe intensity contributions of multiple pixels in the HDR image. Duringaccumulation, near field lighting reconstruction module 220 adjusts theintensity contribution of each pixel to factor out the attenuation oflight along the ray as the light travels from the point light source tothe mirror sphere, using the inverse-square-distance law. Via thistechnique, near field lighting reconstruction module 220 generates areconstructed set of near field positional lights with more accurate,spatially-varying light source intensities during rendering, relative toa conventional environment map that assumes distant light sources.

Because near field lighting reconstruction module 220 generates the nearfield positional lights using all relevant viewpoints, the near fieldpositional lights more faithfully represent the true lighting units inthe scene relative to a conventional environment map. Additionally oralternatively, near field lighting reconstruction module 220 performs aninverse rendering operation without manual intervention, such as byimaging a diffuse object, imaging cross-polarized views of the objectbeing captured, and/or the like. With this inverse rendering approach,near field lighting reconstruction module 220 independently illuminateseach lighting unit one-by-one (and/or in combinations) and adjusting theintensities of the point light sources within the lighting unit to bestfit the captured images. Near field lighting reconstruction module 220repeats this process or each lighting unit.

In operation, polarization modeling module 222 simulates polarized lighttransport including both diffuse reflectance and the portion of thespecular reflectance that remains after polarized light is reflected andfiltered. Polarization modeling module 222 models polarization of lightwith Mueller calculus, which employs Stokes vectors and Muellermatrices. A Stokes vector includes four Stokes parameters, which can beexpressed as s=(s₀, s₁, s₂, s₃). The Stokes parameters include a firstparameter so that expresses intensity, a second parameter s₁ thatexpresses horizontal/vertical polarization, a third parameter s₂ thatexpresses diagonal polarization, and a fourth parameter s₃ thatexpresses circular polarization. A Mueller matrix is a 4×4 matrix thatmodels how polarization of light changes after an incident light beam istransformed, resulting in an emerging light beam. This relationship canbe expressed as s_(e)=M×s_(i), where Stokes vector s_(i) models theincident light, Stokes vector s_(e) models the emergent light, andMueller matrix M models the transformation from the incident light tothe emergent light.

Polarization modeling module 222 simulates each of the lighting units asa lighting unit that casts incident light, represented as s_(i), on apolarization filter, represented by Mueller matrix M₀ that is linearlypolarized in a horizontal direction. As a result, the emergent light islinearly polarized in a horizontal direction, which is expressed asStokes vector s_(a)=M₀×s_(i)=(1, 1, 0, 0). At each point on the objectbeing captured, polarization modeling module 222 determines thedirection of the incident light from the lighting unit via thepolarization filter. Polarization modeling module 222 further determinesthe direction of the outgoing emergent light after specular reflectionoff the surface of the object being captured. Given the direction of theincident light and the direction of the outgoing emergent light,polarization modeling module 222 determines Mueller matrix M₁ thatrepresents the transformation of the incident light from thepolarization filter after specular reflection off the surface of theobject being captured. The reflected light is expressed as Stokes vectors_(b)=M₁×s_(a)=(s₀, s₁, s₂, s₃). Stokes parameter so of Stokes vectors_(b) is referred to herein as the Fresnel gain for the specularcomponent. Polarization modeling module 222 modulates the diffusecomponent of the polarized light with 1 minus the Fresnel gain, or(1−s₀).

In some examples, one or more imaging capture devices are fitted with apolarization filter, represented by Mueller matrix M₂, that is linearlypolarized in a vertical direction, resulting in image capture ofcross-polarized views. In such examples, polarization modeling module222 determines a Stokes vector s_(c)=M₂×M₁×s_(a), where s_(c) representsthe effect of specular reflection of horizontally polarized light offthe surface of the object being captured and subsequently captured by animage capture device with a vertically polarized filter. In this manner,polarization modeling module 222 models the specular reflectancecomponent of polarized light rather than only the diffuse reflectancecomponent of polarized light.

In contrast, if polarization modeling module 222 is not deployed,appearance capture application 210 assumes purely diffuse reflectance ofthe vertically (cross-) polarized views. Further, for image capturedevices that are not fitted with a vertically polarized filter,appearance capture application 210 simply uses the Fresnel curve forp-polarized light with the equations for unpolarized light transport.

In operation, texture space indirect illumination module 224 modelsindirect reflections of light on the surface of the object beingcaptured. The model of indirect reflections of light augments theresults of appearance capture application 210 which models subsurfacescattering using a texture space technique with precomputed visibilityand shadow maps. In this regard, texture space indirect illuminationmodule 224 determines indirect lighting effects by employing a texturespace indirect illumination method for inverse rendering based on MonteCarlo sampling. Texture space is an image space that models the surfaceappearance of a 3D object, such as the object being captured. Texturespace is composed of individual texture elements, referred to herein astexels. Texture space indirect illumination module 224 samples texelcoordinates in UV space of neighboring vertices that contribute toindirect lighting. These neighboring texels effectively act asadditional point lights, where such a neighboring texel reflectsincident light onto other texels. Texture space indirect illuminationmodule 224 fixes the directions of the reflected light rays from theneighboring texels during optimization. In a particular example, texturespace indirect illumination module 224 models indirect reflections oflight that illuminate a current texel 682 ₀ as follows:

At a first step, texture space indirect illumination module 224 projects3D rays from the 3D position of texel μ₀ along random directions overthe hemisphere of the object point being captured that is visible to oneor more imaging devices. For each 3D ray, texture space indirectillumination module 224 determines if the 3D ray intersects the geometryof the object at one or more neighboring 3D points with correspondingtexels μ_(i). Texture space indirect illumination module 224 stores theUV coordinates of these intersections with neighboring texels μ_(i). Ata second step, for each intersection stored in the first step, texturespace indirect illumination module 224 uses the current appearanceparameters at the corresponding neighboring texel μ_(i) to render texelμ_(i) from the viewpoint of the current texel μ₀. Texture space indirectillumination module 224 uses the result of rendering texel μ_(i) as theintensity of indirect lighting incident to texel μ_(i) to constraininverse rendering at texel μ₀. At a third step, texture space indirectillumination module 224 performs an optimization step to update theappearance parameters at texel μ₀, thereby rendering texel μ₀ with bothdirect and indirect lighting. Texture space indirect illumination module224 iterates this process over the texels included in the texture mapand repeats steps 2-3 until convergence is achieved.

The texture space indirect illumination techniques performed by texturespace indirect illumination module 224 can be implemented as anon-invasive module that is incorporated into differentinverse-rendering pipelines, such as appearance capture application 210,to enable indirect illumination and to generate results with a morerealistic appearance. By accounting for indirect reflection of light,texture space indirect illumination module 224 can generate softer andmore realistic rendering of the shadows. As a result, texture spaceindirect illumination module 224 can generate facial appearance maps(such as diffuse and specular albedo maps) that are more complete orwith less overshoot, particularly in areas of concavities on the objectbegin captured. In some examples, texture space indirect illuminationmodule 224 can reduce computational cost by computing indirectillumination only for texels that reside in an area of concavity presentin the object being captured, where the effect of indirect light islikely to be more significant relative to more planar and/or convexportions of the object.

It will be appreciated that the system shown herein is illustrative andthat variations and modifications are possible. Among other things, thetechniques described herein are employed in an appearance capture studiothat is fitted with rectangular LED light sources. Additionally oralternatively, the techniques can be employed with any technicallyfeasible lighting technology and with lighting units of any shape andsize within the scope of the present disclosure. Further, the techniquescan be employed with any number of lighting units and any number ofimage capture devices in any combination. The techniques describedherein include lighting units fitted with horizontally polarized filtersand image capture devices with vertically polarized filters.Additionally or alternatively, the lighting units can be fitted withvertically polarized filters and image capture devices with horizontallypolarized filters. Additionally or alternatively, the lighting units andimage capture devices can be fitted with polarized filters in anyorientation that provides cross-polarized views at the image capturedevices.

FIG. 3 is a schematic diagram of an appearance capture studio 300,according to various embodiments. As shown, the appearance capturestudio 300 includes, without limitation, 32 near field lighting units302(1)-302(32), eight non-polarized image capture devices 304(1)-304(8),four polarized image capture devices 306(1)-306(4), and an object 308,such as human skin or a human head, that is being captured in theappearance capture studio 300.

The 32 near field lighting units 302(1)-302(32) are distributed aroundthe front hemisphere of the object 308 so as to illuminate the facialfeatures of the head. The 32 near field lighting units 302(1)-302(32)are arranged as 16 pairs of 32 near field lighting units 302(1)-302(32).Near field lighting units 302(1)-302(2) form a first pair, near fieldlighting units 302(3)-302(4) form a second pair, and so on, with nearfield lighting units 302(31)-302(32) forming a sixteenth pair. One ormore of the near field lighting units 302(1)-302(32) are fitted withswitchable linearly polarized filters that are engaged and disengaged atdifferent times during appearance capture. For example, the linearlypolarized filters can be engaged when polarization modeling module 222is modeling the specular components of polarized field lighting units.The linearly polarized filters can be disengaged during operations thatdo not used polarized light, such as when texture space indirectillumination module 224 is performing indirect lighting calculations.Additionally or alternatively, a subset of the near field lighting units302(31)-302(32) are fitted with linearly polarized filters and adifferent subset of the near field lighting units 302(31)-302(32) areunpolarized field lighting units. In such cases, the polarized fieldlighting units and the unpolarized field lighting units can beilluminated or extinguished as needed at different times duringappearance capture.

The eight non-polarized image capture devices 304(1)-304(8) and fourpolarized image capture devices 306(1)-306(4) are distributed around thefront hemisphere of the object 308 so as to capture the appearance ofthe facial features of the head as illuminated by near field lightingunits 302(1)-302(32). In some embodiments, the eight non-polarized imagecapture devices 304(1)-304(8) can be replaced by eightparallel-polarized image capture devices that capture complementaryreflectance data, relative to the cross-polarized image capture devices.The eight non-polarized image capture devices 304(1)-304(8) and fourpolarized image capture devices 306(1)-306(4) are arranged into fourtriplets, where each triplet of image capture devices includes twonon-polarized image capture devices 304 and one polarized image capturedevice 306. More specifically, non-polarized image capture devices304(1)-304(2) and polarized image capture device 306(1) form a firsttriplet of image capture devices. Non-polarized image capture devices304(3)-304(4) and polarized image capture device 306(2) form a secondtriplet of image capture devices, and so on. Each pair of non-polarizedimage capture devices 304 captures full facial reflectance and is alsoused for stereo reconstruction. Each polarized image capture device 306is cross-polarized with respect to the polarization filters on the nearfield lighting units 302(31)-302(32). Each polarized image capturedevice 306 mainly captures diffuse reflection as well as specularreflection that remains in the polarized image views.

FIGS. 4A-4B illustrates techniques for reconstructing a near fieldlighting unit 402, according to various embodiments. As shown in FIG.4A, near field lighting reconstruction module 220 projects 3D rays 414from each of three image capture devices 404 onto a mirror sphere 408.More particularly, near field lighting reconstruction module 220 shownin FIG. 2 projects 3D rays 414(1), 414(2), and 414(3) from each of imagecapture devices 404(1), 404(2), and 404(3) respectively. Based on theposition and size of the mirror sphere 408, near field lightingreconstruction module 220 determines the incident angle of each of the3D rays 414(1), 414(2), and 414(3) as well as the reflection angles forcorresponding reflected 3D rays 424(1), 424(2), and 424(3). Inparticular, near field lighting reconstruction module 220 determinesthat 3D ray 414(1) reflects off the surface of the mirror sphere,resulting in 3D ray 424(1). Likewise, near field lighting reconstructionmodule 220 determines that 3D ray 414(2) reflects off the surface of themirror sphere, resulting in 3D ray 424(2), and that 3D ray 414(3)reflects off the surface of the mirror sphere, resulting in 3D ray424(3). Near field lighting reconstruction module 220 determines wherethe reflected 3D rays 424(1), 424(2), and 424(3) intersect with the nearfield lighting units 402.

As shown, reflected 3D rays 424(1), 424(2), and 424(3) intersect nearfield lighting unit 402-A at point 412-A. Near field lighting unit 402-Ais shown in front facing view in FIG. 4B as near field lighting unit402-B. Near field lighting unit 402-B is modeled as an array of pointlight sources, where all of the reflected 3D rays 424(1), 424(2), and424(3) intersect near field lighting unit 402-B at point light source412-B. In alternative embodiments, one or more near field lightingunits, such as near field lighting unit 402-B, can be modeled as anyarray of point light sources, such as a 1D linear array of point lightsources, a 2D grid arranged as multiple rows of point light sources, anarbitrary 2D shape of point light sources, and/or the like. As a result,near field lighting reconstruction module 220 accumulates the intensitycontribution of reflected 3D rays 424(1), 424(2), and 424(3) and storesthe intensity contribution as part of the illumination associated withpoint light source 412-B. As a result, point light source 412-B, as wellas all point light sources 412 for all near field lighting units 402, isbased on the positions and orientations of the near field lighting units402, the image capture devices 404, and the mirror sphere 408.

FIG. 5 illustrates techniques for modeling specular components of apolarized lighting unit, according to various embodiments. As shown, alighting unit 502 generates light and is modeled with Stokes vectors_(i). Lighting unit 502 generates incident light that intersects withpolarization filter 504 that is modeled with Mueller matrix M₀. Incidentlight from lighting unit 502 that intersects with polarization filter504 is transformed by Mueller matrix M₀. If polarization filter 504 islinearly polarized in a horizontal direction, then the resulting lightthat passes through polarization filter 504 is modeled by the Stokesvector s_(a)=M₀×s_(i)=(1, 1, 0, 0). The resulting light transmitted viapolarization filter 504 strikes the object 506 being captured. Given thedirection of the incident light from polarization filter 504 and thedirection of the outgoing emergent light reflecting off of the object506 being captured, polarization modeling module 222 determines Muellermatrix M₁ that represents the transformation of the incident light frompolarization filter 504 after specular reflection off the surface of theobject 506 being captured. The reflected light is expressed as Stokesvector s_(b)=M₁×s_(a)=(s₀, s₁, s₂, s₃).

If image capture device 508 is fitted with a second polarization filter,represented by Mueller matrix M₂, that is linearly cross-polarized withrespect to polarization filter 504, then image capture device 508captures cross-polarized images. In such examples, polarization modelingmodule 222 determines a Stokes vector s_(c)=M₂×M₁×s_(a), where s_(c)represents the effect of specular reflection of horizontally polarizedlight off the surface of the object being captured and subsequentlycaptured by an image capture device with a vertically polarized filter.In this manner, polarization modeling module 222 models the specularreflectance component of polarized light in cross-polarized views aswell as the diffuse reflectance component of polarized light.

FIG. 6 illustrates techniques for modeling indirect lighting of anobject 608 being captured, according to various embodiments. As shown,the surface of the object 608 includes a texel μ₀ 602 that isilluminated by direct light from one or more lighting units (not shown)and indirect light from neighboring texels μ₁ 606(1), μ₂ 606(2), and μ₃606(3). To model indirect lighting of object 608, texture space indirectillumination module 224 projects 3D rays 604(1)-604(3) from the 3Dposition of texel μ₀ along random directions over the hemisphere of theobject 608 being captured that is visible to one or more imagingdevices. For each 3D ray, such as 3D rays 604(1)-604(3), texture spaceindirect illumination module 224 determines if the 3D ray intersects thegeometry of the object 608 at one or more neighboring texels. As shown,3D rays 604(1)-604(3) intersect neighboring texels μ₁ 606(1), μ₂ 606(2),and μ₃ 606(3), respectively. Texture space indirect illumination module224 stores the UV coordinates of these intersections with neighboringtexels μ₁ 606(1), μ₂ 606(2), and μ₃ 606(3).

For each stored intersection, texture space indirect illumination module224 uses the current appearance parameters at the correspondingneighboring texels μ₁ 606(1), μ₂ 606(2), and μ₃ 606(3) to render texelsμ₁ 606(1), μ₂ 606(2), and μ₃ 606(3), respectively, from the viewpoint ofthe current texel μ₀ 602. Texture space indirect illumination module 224uses the result of rendering neighboring texels μ₁ 606(1), μ₂ 606(2),and μ₃ 606(3) as contributions to the intensity of indirect lightingfrom neighboring texels μ₁ 606(1), μ₂ 606(2), and μ₃ 606(3),respectively, to constrain inverse rendering at texel μ₀ 602. Texturespace indirect illumination module 224 performs an optimization step toupdate the appearance parameters at texel μ₀ 602, thereby renderingtexel μ₀ 602 with both direct and indirect lighting.

FIG. 7 is a flow diagram of method steps for reconstructing near fieldlighting, according to various embodiments. Although the method stepsare described in conjunction with the systems of FIGS. 1-6 , persons ofordinary skill in the art will understand that any system configured toperform the method steps, in any order, is within the scope of thepresent disclosure.

As shown, a method 700 begins at step 702, where a CPU 102 executing anear field lighting reconstruction module 220 determines at least one ofa 3D position or a 3D orientation of a lighting unit based on aplurality of captured images of a mirror sphere. CPU 102 determines theposition and orientation of each lighting unit. To locate each lightingunit in 3D space, CPU 102 identifies and annotates the corner verticesof each lighting unit as the corner vertices appear on the multi-viewimages of the mirror sphere, as captured by the image capture devices.For each identified and annotated corner vertex, CPU 102 projects 3Drays from each image capture device through the identified and annotatedcorner vertex and reflects the 3D rays off of the mirror sphere. CPU 102sets the position of the current 3D corner vertex as the position in 3Dspace that is closest to all reflected rays. CPU 102 repeats thisprocess for all remaining corner vertices for each of the lightingunits. In addition, CPU 102 determines the orientation of each lightingunit. Based on the 3D positions of the corner vertices of the lightingunit, CPU 102 determines the plane in 3D space that intersects thecorner vertices. Based on this plane, CPU 102 computes a normal vectorof the plane. CPU 102 selects the direction of the normal vector that istowards the center of the mirror sphere. In some examples, CPU 102determines the orientation of each lighting unit by weakly enforcing aregularization constraint that orients the normal vector of the plane ofthe lighting unit towards the center of the mirror sphere, whileaccounting for potential inaccuracies in the annotations of the cornervertices.

At step 704, for each point light source in a plurality of point lightsources included in the lighting unit, CPU 102 determines an intensityassociated with the point light source. In so doing, CPU 102 samples aset of point light sources inside each lighting unit. CPU 102 capturesan HDR image of the mirror sphere with an unpolarized image capturedevice. For each pixel in this HDR image, CPU 102 projects a 3D raytowards the mirror sphere and assigns the observed pixel intensity tothe point light source that is nearest to the reflected ray. Each pointlight source can accumulate the intensity contributions of multiplepixels in the HDR image. During accumulation, CPU 102 adjusts theintensity contribution of each pixel to factor out the attenuation oflight along the ray as the light travels from the point light source tothe mirror sphere, using the inverse-square-distance law. Via thistechnique, CPU 102 generates a reconstructed set of near fieldpositional lighting units with more accurate, spatially-varying lightsource intensities during rendering, relative to a conventionalenvironment map that assumes distant lighting units.

At step 706, CPU 102 captures appearance data of an object, such ashuman skin or a human head, where the object is illuminated by thelighting unit. CPU 102 captures the appearance based on the intensitiesdetermined at steps 702 and 704.

At step 708, CPU 102 renders an image of the object being captured basedon the appearance data captured at step 706 and the intensitiesassociated with each point light source determined at steps 702 and 704.As a result, the image of the object being captured is more accuratebecause the image is based on a more realistic model of near fieldlighting units relative to conventional techniques. In some examples,CPU 102 performing an inverse rendering operation without manualintervention, such as by imaging a diffuse object, imagingcross-polarized views of the object being captured, and/or the like.With this inverse rendering approach, CPU 102 independently illuminateseach lighting unit one-by-one (and/or in combinations) and adjusts theintensities of the point light sources within the lighting unit to bestfit the captured images. The method 700 then terminates.

FIG. 8 is a flow diagram of method steps for modeling specularreflection of polarized light, according to various embodiments.Although the method steps are described in conjunction with the systemsof FIGS. 1-6 , persons of ordinary skill in the art will understand thatany system configured to perform the method steps, in any order, iswithin the scope of the present disclosure.

As shown, a method 800 begins at step 802, where, for a first point onan object being captured, such as human skin or a human head, a CPU 102executing a polarization modeling module 222 determines an incidentdirection of polarized light emanating from the lighting unit andintersecting the first point. CPU 102 simulates each of the lightingunits as a lighting unit that casts incident light, represented ass_(i), on a polarization filter, represented by Mueller matrix M₀ thatis linearly polarized in a horizontal direction. As a result, theemergent light is linearly polarized in a horizontal direction, which isexpressed as Stokes vector s_(a)=M₀×s_(i)=(1, 1, 0, 0). At each point onthe object being captured, CPU 102 determines the direction of theincident light from the lighting unit via the polarization filter.

At step 804, the CPU 102 determines an outgoing direction of thepolarized light after reflecting off the first point. In addition todetermining an incident direction of polarized light emanating from thelighting unit, the CPU 102 further determines the direction of theoutgoing emergent light after specular reflection off the surface of theobject being captured. Given the direction of the incident light and thedirection of the outgoing emergent light, CPU 102 determines Muellermatrix M₁ that represents the transformation of the incident light fromthe polarization filter after specular reflection off the surface of theobject being captured.

At step 806, the CPU 102 determines a gain for a specular component ofthe polarized light based on the incident direction and the outgoingdirection. The reflected light determined at step 804 is expressed asStokes vector s_(b)=M₁×s_(a)=(s₀, s₁, s₂, s₃). Stokes parameter s₀ ofStokes vector s_(b) is referred to herein as the Fresnel gain for thespecular component.

At step 808, CPU 102 modulates a diffuse component of the polarizedlight with the gain. CPU 102 modulates the diffuse component of thepolarized light with 1 minus the Fresnel gain, or (1−s₀). In thismanner, CPU 102 models specular reflectance component of polarized lightrather than only the diffuse reflectance component of polarized light.The method 800 then terminates.

FIG. 9 is a flow diagram of method steps for determining indirectlighting effects during appearance capture, according to variousembodiments. Although the method steps are described in conjunction withthe systems of FIGS. 1-6 , persons of ordinary skill in the art willunderstand that any system configured to perform the method steps, inany order, is within the scope of the present disclosure.

As shown, a method 900 begins at step 902, where a CPU 102 executing atexture space indirect illumination module 224 determines texturecoordinates of a vector originating from a first texel where the vectorintersects a second texel. CPU 102 projects 3D rays from the 3D positionof texel μ₀ along random directions over the hemisphere of the objectbeing captured that is visible to one or more imaging devices. For each3D ray, CPU 102 determines if the 3D ray intersects the geometry of theobject at one or more neighboring texels μ_(i). CPU 102 stores the UVcoordinates of these intersections with neighboring texels μ_(i).

At step 904, CPU 102 renders the second texel from the viewpoint of thefirst texel based on appearance data at the second texel. For eachintersection stored in at step 902, CPU 102 uses the current appearanceparameters at the corresponding neighboring texel μ_(i) to render texelμ_(i) from the viewpoint of the current texel μ₀.

At step 906, the CPU 102, based on the rendering of the second texel,determines an indirect lighting intensity incident to the first texelfrom the second texel. CPU 102 uses the result of rendering texel μ_(i)at step 904 as the intensity of indirect lighting incident to texelμ_(i) to constrain inverse rendering at texel μ₀.

At step 908, the CPU 102 updates appearance data at the first texelbased on a direct lighting intensity and the indirect lightingintensity. CPU 102 performs an optimization step to update theappearance parameters at texel μ₀, thereby rendering texel μ₀ with bothdirect and indirect lighting.

At step 910, CPU 102 renders the first texel based on the updatedappearance data at the first texel. By accounting for indirectreflection of light, CPU 102 can generate softer and more realisticrendering of the shadows. As a result, CPU 102 can generate facialappearance maps (such as diffuse and specular albedo maps) that are morecomplete or with less overshoot, particularly in areas of concavities onthe object begin captured. The method 900 then terminates.

In sum, various embodiments are directed to techniques for capturingfacial appearance data for the purpose of realistic rendering of humanfaces. The embodiments include improved lighting models for appearancecapture that more accurately represent the lighting, while minimallyincreasing computational cost. The disclosed techniques can be employedin any combination.

In a first technique, referred to as “near field light reconstruction,”a computer system uses multi-view images and triangulation to preciselymeasure the 3D locations of lighting unit for high-fidelity appearancecapture. With this technique, the geometry of each lighting unit isrepresented as a rectangular 3D surface with known width and height(from manual measurements) and with position and orientation informationin 3D space that is determined from multi-view images of a mirrorsphere.

In a second technique, referred to as “polarization modeling,” acomputer system models a specular component in cross-polarized viewsrather than assuming purely diffuse reflectance for cross-polarizedviews. In a cross-polarized view, the lighting unit is polarized in onedirection and the cameras are fitted with filters that are polarized ina different cross-polarized direction. Although most reflectance in across-polarized view is diffuse reflectance, the second technique modelsthe remaining specular reflectance.

In a third technique, referred to as texture space indirectillumination, a computer system models indirect lighting that resultsfrom light reflecting off one portion of the face and illuminating adifferent part of the, such as when light reflects off the inner cheekand partially illuminates the nose. This technique can be readilyincorporated into different inverse-rendering pipelines.

At least one technical advantage of the disclosed techniques relative tothe prior art is that, with the disclosed near field lightreconstruction, lighting is more accurately modeled for multiple nearfield lighting units. As a result, a field of near positional lights isreconstructed that more faithfully represents the true lights in thescene as compared to a conventional distant environment map. Thedisclosed polarization modeling more accurately models specularreflectance present in polarized lighting units, thereby improvingappearance and reducing rendering error relative to techniques thatmodel only the diffuse reflectance from polarized lighting units. Thedisclosed texture space indirect illumination more accurately modelsfacial characteristics where light reflects off one portion of a facialmodel and strikes another portion of the facial model. Another technicaladvantage relative to the prior art is that the disclosed techniques,whether employed individually or in combination, provide improvedaccuracy and realism when rendering objects from appearance capture datawith only modest increases in computational cost relative to priortechniques. The disclosed techniques account for the impact on inverserendering quality when departing from idealized conditions, leading tomore realistic models that more accurately capture and render facialappearance. These advantages represent one or more technologicalimprovements over prior art approaches.

1. In some embodiments, a computer-implemented method for rendering anobject from captured appearance data comprises: determining at least oneof a three-dimensional (3D) position or a 3D orientation of a lightingunit based on a plurality of captured images of a mirror sphere; foreach point light source in a plurality of point light sources includedin the lighting unit, determining an intensity associated with the pointlight source; capturing appearance data of the object, where the objectis illuminated by the lighting unit; and rendering an image of theobject based on the appearance data and the intensities associated witheach point light source in the plurality of point light sources.

2. The computer-implemented method according to clause 1, furthercomprising: for a first point on the object: determining an incidentdirection of polarized light emanating from the lighting unit andintersecting the first point; determining an outgoing direction of thepolarized light after reflecting off the first point; determining a gainfor a specular component of the polarized light based on the incidentdirection and the outgoing direction; and modulating a diffuse componentof the polarized light with the gain.

3. The computer-implemented method according to clause 1 or clause 2,wherein: the polarized light emanating from the lighting unit ispolarized in a first direction, an image capture device configured tocapture an image of the object is fitted with a filter that is polarizedin a second direction, and the diffuse component of the polarized lightis further modulated based on a model of the image capture device.

4. The computer-implemented method according to any of clauses 1-3,wherein determining the 3D position of the lighting unit comprises: foreach image capture device included in a plurality of image capturedevices: identifying a corner vertex of the lighting unit in an image ofthe mirror sphere captured by the image capture device; projecting a 3Dray from the image capture device towards the mirror sphere, andreflecting the 3D ray off the mirror sphere to generate a reflected 3Dray; and setting the 3D position of the lighting unit as a 3D positionthat is closest to the reflected 3D rays from the plurality of imagecapture devices.

5. The computer-implemented method according to any of clauses 1-4,wherein setting the 3D position of the lighting unit comprisesregressing the reflected 3D rays to the 3D position of the lighting unitby applying least squares regression to the reflected 3D rays.

6. The computer-implemented method according to any of clauses 1-5,wherein determining the 3D orientation of the lighting unit comprises:determining 3D positions for a set of corner vertices of the lightingunit; determining a plane in 3D space that intersects the set of cornervertices of the lighting unit; computing a normal vector of the plane;and selecting a direction of the normal vector that is towards a centerof the mirror sphere.

7. The computer-implemented method according to any of clauses 1-6,wherein determining the intensity associated with the point light sourcecomprises: projecting a first 3D ray from a first pixel in a firstcaptured image included in the plurality of captured images towards themirror sphere; reflecting the first 3D ray off the mirror sphere togenerate a first reflected 3D ray; determining that the first reflected3D ray intersects a first point light source in the plurality of pointlight sources; and setting the intensity associated with the first pointlight source to an intensity contribution of the first pixel.

8. The computer-implemented method according to any of clauses 1-7,wherein determining the intensity associated with the point light sourcefurther comprises adjusting the intensity contribution of the firstpixel based on an attenuation of light along the first 3D ray as lightfrom the first 3D ray travels from the first point light source to themirror sphere.

9. The computer-implemented method according to any of clauses 1-8,wherein the attenuation of the light along the first 3D ray is based onan inverse-square-distance law.

10. The computer-implemented method according to any of clauses1-9,wherein determining the intensity associated with the point lightsource further comprises: projecting a second 3D ray from a second pixelin the first captured image towards the mirror sphere; reflecting thesecond 3D ray off the mirror sphere to generate a second reflected 3Dray; determining that the second reflected 3D ray intersects the firstpoint light source; accumulating the intensity associated with the firstpoint light source with an intensity contribution of the second pixel togenerate an accumulated intensity; and setting the intensity associatedwith the first point light source to the accumulated intensity.

11. The computer-implemented method according to any of clauses 1-10,further comprising performing an inverse rendering operation to adjustthe intensity associated with the first point light source to best fitthe first captured image.

12. In some embodiments, one or more non-transitory computer-readablemedia store program instructions that, when executed by one or moreprocessors, cause the one or more processors to perform steps of:determining at least one of a three-dimensional (3D) position or a 3Dorientation of a lighting unit based on a plurality of captured imagesof a mirror sphere; for each point light source in a plurality of pointlight sources included in the lighting unit, determining an intensityassociated with the point light source; capturing appearance data of anobject, where the object is illuminated by the lighting unit; andrendering an image of the object based on the appearance data and theintensities associated with each point light source in the plurality ofpoint light sources.

13. The one or more non-transitory computer-readable media according toclause 1, wherein determining the 3D position of the lighting unitcomprises: for each image capture device included in a plurality ofimage capture devices: identifying a corner vertex of the lighting unitin an image of the mirror sphere captured by the image capture device;projecting a 3D ray from the image capture device towards the mirrorsphere, and reflecting the 3D ray off the mirror sphere to generate areflected 3D ray; and setting the 3D position of the lighting unit as a3D position that is closest to the reflected 3D rays from the pluralityof image capture devices.

14. The one or more non-transitory computer-readable media according toclause 12 or clause 13, wherein setting the 3D position of the lightingunit comprises regressing the reflected 3D rays to the 3D position ofthe lighting unit by applying least squares regression to the reflected3D rays.

15. The one or more non-transitory computer-readable media according toany of clauses 12-14, wherein determining the 3D orientation of thelighting unit comprises: determining 3D positions for a set of cornervertices of the lighting unit; determining a plane in 3D space thatintersects the set of corner vertices of the lighting unit; computing anormal vector of the plane; and selecting a direction of the normalvector that is towards the center of the mirror sphere.

16. The one or more non-transitory computer-readable media according toany of clauses 12-15, wherein determining the intensity associated withthe point light source comprises: projecting a first 3D ray from a firstpixel in a first captured image included in the plurality of capturedimages towards the mirror sphere; reflecting the first 3D ray off themirror sphere to generate a first reflected 3D ray; determining that thefirst reflected 3D ray intersects a first point light source in theplurality of point light sources; and setting the intensity associatedwith the first point light source to an intensity contribution of thefirst pixel.

17. The one or more non-transitory computer-readable media according toany of clauses 12-16, wherein determining the intensity associated withthe point light source further comprises adjusting the intensitycontribution of the first pixel based on an attenuation of light alongthe first 3D ray as light from the first 3D ray travels from the firstpoint light source to the mirror sphere.

18. The one or more non-transitory computer-readable media according toany of clauses 12-17, wherein the attenuation of the light along thefirst 3D ray is based on an inverse-square-distance law.

19. The one or more non-transitory computer-readable media according toany of clauses 12-18, wherein determining the intensity associated withthe point light source further comprises: projecting a second 3D rayfrom a second pixel in the first captured image towards the mirrorsphere; reflecting the second 3D ray off the mirror sphere to generate asecond reflected 3D ray; determining that the second reflected 3D rayintersects the first point light source; accumulating the intensityassociated with the first point light source with an intensitycontribution of the second pixel to generate an accumulated intensity;and setting the intensity associated with the first point light sourceto the accumulated intensity.

20. In some embodiments, a system comprises: one or more memoriesstoring instructions; and one or more processors coupled to the one ormore memories and, when executing the instructions: determine at leastone of a three-dimensional (3D) position or a 3D orientation of alighting unit based on a plurality of captured images of a mirrorsphere; for each point light source in a plurality of point lightsources included in the lighting unit, determine an intensity associatedwith the point light source; capture appearance data of an object, wherethe object is illuminated by the lighting unit; and render an image ofthe object based on the appearance data and the intensities associatedwith each point light source in the plurality of point light sources.

21. In some embodiments, a computer-implemented method for rendering anobject from captured appearance data comprising a plurality of texelscomprises: determining texture coordinates of a first vector originatingfrom a first texel where the first vector intersects a second texel;rendering the second texel from a viewpoint of the first texel based onappearance data at the second texel; based on the rendering of thesecond texel, determining an indirect lighting intensity incident to thefirst texel from the second texel; updating appearance data at the firsttexel based on a direct lighting intensity and the indirect lightingintensity incident to the first texel from the second texel; andrendering the first texel based on the updated appearance data at thefirst texel.

22. The computer-implemented method according to clause 21, furthercomprising: for a first point on the object: determining an incidentdirection of polarized light emanating from a lighting unit andintersecting the first point; determining an outgoing direction of thepolarized light after reflecting off the first point; determining a gainfor a specular component of the polarized light based on the incidentdirection and the outgoing direction; and modulating a diffuse componentof the polarized light with the gain.

23. The computer-implemented method according to clause 21 or clause 22,wherein: the polarized light emanating from the lighting unit ispolarized in a first direction, an image capture device configured tocapture an image of the object is fitted with a filter that is polarizedin a second direction, and the diffuse component of the polarized lightis further modulated based on a model of the image capture device.

24. The computer-implemented method according to any of clauses 21-23,wherein determining texture coordinates of the first vector originatingfrom the first texel where the first vector intersects the second texelcomprises: projecting a plurality of 3D rays from a 3D position of thefirst texel; determining intersection coordinates where a first 3D rayincluded in the plurality of 3D rays intersects the second texel; andsetting the intersection coordinates as the texture coordinates of thefirst vector.

25. The computer-implemented method according to any of clauses 21-24,wherein the plurality of 3D rays is projected along random directionsover a hemisphere of the object that is visible to one or more imagingdevices.

26. The computer-implemented method according to any of clauses 21-25,further comprising: determining texture coordinates of a second vectororiginating from the first texel where the second vector intersects athird texel; rendering the third texel from a viewpoint of the firsttexel based on appearance data at the third texel; based on therendering of the third texel, determining an indirect lighting intensityincident to the first texel from the third texel; and updatingappearance data at the first texel further based on the indirectlighting intensity incident to the first texel from the third texel.

27. The computer-implemented method according to any of clauses 21-26,wherein the texture coordinates comprise coordinates in UV space.

28. The computer-implemented method according to any of clauses 21-27,further comprising, prior to determining texture coordinates of thefirst vector originating from the first texel where the first vectorintersects the second texel, determining that the first texel resides inan area of concavity on the object.

29. In some embodiments, one or more non-transitory computer-readablemedia store program instructions that, when executed by one or moreprocessors, cause the one or more processors to perform steps of:determining texture coordinates of a first vector originating from afirst texel where the first vector intersects a second texel; renderingthe second texel from a viewpoint of the first texel based on appearancedata at the second texel; based on the rendering of the second texel,determining an indirect lighting intensity incident to the first texelfrom the second texel; updating appearance data at the first texel basedon a direct lighting intensity and the indirect lighting intensity; andrendering the first texel based on the updated appearance data at thefirst texel.

30. The one or more non-transitory computer-readable media according toclause 29, wherein the steps further comprise: for a first point on anobject that includes the first texel: determining an incident directionof polarized light emanating from a lighting unit and intersecting thefirst point; determining an outgoing direction of the polarized lightafter reflecting off the first point; determining a gain for a specularcomponent of the polarized light based on the incident direction and theoutgoing direction; and modulating a diffuse component of the polarizedlight with the gain.

31. The one or more non-transitory computer-readable media according toclause 29 or clause 30, wherein: the polarized light emanating from thelighting unit is polarized in a first direction, an image capture deviceconfigured to capture an image of the object is fitted with a filterthat is polarized in a second direction, and the diffuse component ofthe polarized light is further modulated based on a model of the imagecapture device.

32. The one or more non-transitory computer-readable media according toany of clauses 29-31, wherein determining texture coordinates of thefirst vector originating from the first texel where the first vectorintersects the second texel comprises: projecting a plurality of 3D raysfrom a 3D position of the first texel; determining intersectioncoordinates where a first 3D ray included in the plurality of 3D raysintersects the second texel; and setting the intersection coordinates asthe texture coordinates of the first vector.

33. The one or more non-transitory computer-readable media according toany of clauses 29-32, wherein the plurality of 3D rays is projectedalong random directions over a hemisphere of the object that is visibleto one or more imaging devices.

34. The one or more non-transitory computer-readable media according toany of clauses 29-33, wherein the steps further comprise: determiningtexture coordinates of a second vector originating from the first texelwhere the second vector intersects a third texel; rendering the thirdtexel from a viewpoint of the first texel based on appearance data atthe third texel; based on the rendering of the third texel, determiningan indirect lighting intensity incident to the first texel from thethird texel; and updating appearance data at the first texel furtherbased on the indirect lighting intensity incident to the first texelfrom the third texel.

35. The one or more non-transitory computer-readable media according toany of clauses 29-34, wherein the texture coordinates comprisecoordinates in UV space.

36. The one or more non-transitory computer-readable media according toany of clauses 29-35, wherein the steps further comprise, prior todetermining texture coordinates of the first vector originating from thefirst texel where the first vector intersects the second texel,determining that the first texel resides in an area of concavity on theobject.

37. In some embodiments, a system, comprises: one or more memoriesstoring instructions; and one or more processors coupled to the one ormore memories and, when executing the instructions: determine texturecoordinates of a first vector originating from a first texel where thefirst vector intersects a second texel; render the second texel from aviewpoint of the first texel based on appearance data at the secondtexel; based on the rendering of the second texel, determine an indirectlighting intensity incident to the first texel from the second texel;update appearance data at the first texel based on a direct lightingintensity and the indirect lighting intensity; and render the firsttexel based on the updated appearance data at the first texel.

38. The system according to clause 37, wherein, to determine texturecoordinates of the first vector originating from the first texel wherethe first vector intersects the second texel, the one or more processorsfurther: project a plurality of 3D rays from a 3D position of the firsttexel; determine intersection coordinates where a first 3D ray includedin the plurality of 3D rays intersects the second texel; and set theintersection coordinates as the texture coordinates of the first vector.

39. The system according to clause 37 or clause 38, wherein theplurality of 3D rays is projected along random directions over ahemisphere of the object that is visible to one or more imaging devices.

40. The system according to any of clauses 37-39, wherein the one ormore processors further: determine texture coordinates of a secondvector originating from the first texel where the second vectorintersects a third texel; render the third texel from a viewpoint of thefirst texel based on appearance data at the third texel; based on therendering of the third texel, determine an indirect lighting intensityincident to the first texel from the third texel; and update appearancedata at the first texel further based on the indirect lighting intensityincident to the first texel from the third texel.

Any and all combinations of any of the claim elements recited in any ofthe claims and/or any elements described in this application, in anyfashion, fall within the contemplated scope of the present disclosureand protection.

The descriptions of the various embodiments have been presented forpurposes of illustration, but are not intended to be exhaustive orlimited to the embodiments disclosed. Many modifications and variationswill be apparent to those of ordinary skill in the art without departingfrom the scope and spirit of the described embodiments.

Aspects of the present embodiments may be embodied as a system, method,or computer program product. Accordingly, aspects of the presentdisclosure may take the form of an entirely hardware embodiment, anentirely software embodiment (including firmware, resident software,micro-code, etc.) or an embodiment combining software and hardwareaspects that may all generally be referred to herein as a “module” or“system.” Furthermore, aspects of the present disclosure may take theform of a computer program product embodied in one or more computerreadable medium(s) having computer readable program code embodiedthereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

Aspects of the present disclosure are described above with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of thedisclosure. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, enable the implementation of the functions/acts specified inthe flowchart and/or block diagram block or blocks. Such processors maybe, without limitation, general purpose processors, special-purposeprocessors, application-specific processors, or field-programmable gatearrays.

The flowchart and block diagrams in the figures illustrate thearchitecture, functionality, and operation of possible implementationsof systems, methods, and computer program products according to variousembodiments of the present disclosure. In this regard, each block in theflowchart or block diagrams may represent a module, segment, or portionof code, which comprises one or more executable instructions forimplementing the specified logical function(s). It should also be notedthat, in some alternative implementations, the functions noted in theblock may occur out of the order noted in the figures. For example, twoblocks shown in succession may, in fact, be executed substantiallyconcurrently, or the blocks may sometimes be executed in the reverseorder, depending upon the functionality involved. It will also be notedthat each block of the block diagrams and/or flowchart illustration, andcombinations of blocks in the block diagrams and/or flowchartillustration, can be implemented by special purpose hardware-basedsystems that perform the specified functions or acts, or combinations ofspecial purpose hardware and computer instructions.

While the preceding is directed to embodiments of the presentdisclosure, other and further embodiments of the disclosure may bedevised without departing from the basic scope thereof, and the scopethereof is determined by the claims that follow.

What is claimed is:
 1. A computer-implemented method for rendering anobject from captured appearance data, the method comprising: determiningat least one of a three-dimensional (3D) position or a 3D orientation ofa lighting unit based on a plurality of captured images of a mirrorsphere; for each point light source in a plurality of point lightsources included in the lighting unit, determining an intensityassociated with the point light source; capturing appearance data of theobject, where the object is illuminated by the lighting unit; andrendering an image of the object based on the appearance data and theintensities associated with each point light source in the plurality ofpoint light sources.
 2. The computer-implemented method of claim 1,further comprising: for a first point on the object: determining anincident direction of polarized light emanating from the lighting unitand intersecting the first point; determining an outgoing direction ofthe polarized light after reflecting off the first point; determining again for a specular component of the polarized light based on theincident direction and the outgoing direction; and modulating a diffusecomponent of the polarized light with the gain.
 3. Thecomputer-implemented method of claim 2, wherein: the polarized lightemanating from the lighting unit is polarized in a first direction, animage capture device configured to capture an image of the object isfitted with a filter that is polarized in a second direction, and thediffuse component of the polarized light is further modulated based on amodel of the image capture device.
 4. The computer-implemented method ofclaim 1, wherein determining the 3D position of the lighting unitcomprises: for each image capture device included in a plurality ofimage capture devices: identifying a corner vertex of the lighting unitin an image of the mirror sphere captured by the image capture device;projecting a 3D ray from the image capture device towards the mirrorsphere, and reflecting the 3D ray off the mirror sphere to generate areflected 3D ray; and setting the 3D position of the lighting unit as a3D position that is closest to the reflected 3D rays from the pluralityof image capture devices.
 5. The computer-implemented method of claim 4,wherein setting the 3D position of the lighting unit comprisesregressing the reflected 3D rays to the 3D position of the lighting unitby applying least squares regression to the reflected 3D rays.
 6. Thecomputer-implemented method of claim 1, wherein determining the 3Dorientation of the lighting unit comprises: determining 3D positions fora set of corner vertices of the lighting unit; determining a plane in 3Dspace that intersects the set of corner vertices of the lighting unit;computing a normal vector of the plane; and selecting a direction of thenormal vector that is towards a center of the mirror sphere.
 7. Thecomputer-implemented method of claim 1, wherein determining theintensity associated with the point light source comprises: projecting afirst 3D ray from a first pixel in a first captured image included inthe plurality of captured images towards the mirror sphere; reflectingthe first 3D ray off the mirror sphere to generate a first reflected 3Dray; determining that the first reflected 3D ray intersects a firstpoint light source in the plurality of point light sources; and settingthe intensity associated with the first point light source to anintensity contribution of the first pixel.
 8. The computer-implementedmethod of claim 7, wherein determining the intensity associated with thepoint light source further comprises adjusting the intensitycontribution of the first pixel based on an attenuation of light alongthe first 3D ray as light from the first 3D ray travels from the firstpoint light source to the mirror sphere.
 9. The computer-implementedmethod of claim 8, wherein the attenuation of the light along the first3D ray is based on an inverse-square-distance law.
 10. Thecomputer-implemented method of claim 7, wherein determining theintensity associated with the point light source further comprises:projecting a second 3D ray from a second pixel in the first capturedimage towards the mirror sphere; reflecting the second 3D ray off themirror sphere to generate a second reflected 3D ray; determining thatthe second reflected 3D ray intersects the first point light source;accumulating the intensity associated with the first point light sourcewith an intensity contribution of the second pixel to generate anaccumulated intensity; and setting the intensity associated with thefirst point light source to the accumulated intensity.
 11. Thecomputer-implemented method of claim 7, further comprising performing aninverse rendering operation to adjust the intensity associated with thefirst point light source to best fit the first captured image.
 12. Oneor more non-transitory computer-readable media storing programinstructions that, when executed by one or more processors, cause theone or more processors to perform steps of: determining at least one ofa three-dimensional (3D) position or a 3D orientation of a lighting unitbased on a plurality of captured images of a mirror sphere; for eachpoint light source in a plurality of point light sources included in thelighting unit, determining an intensity associated with the point lightsource; capturing appearance data of an object, where the object isilluminated by the lighting unit; and rendering an image of the objectbased on the appearance data and the intensities associated with eachpoint light source in the plurality of point light sources.
 13. The oneor more non-transitory computer-readable media of claim 12, whereindetermining the 3D position of the lighting unit comprises: for eachimage capture device included in a plurality of image capture devices:identifying a corner vertex of the lighting unit in an image of themirror sphere captured by the image capture device; projecting a 3D rayfrom the image capture device towards the mirror sphere, and reflectingthe 3D ray off the mirror sphere to generate a reflected 3D ray; andsetting the 3D position of the lighting unit as a 3D position that isclosest to the reflected 3D rays from the plurality of image capturedevices.
 14. The one or more non-transitory computer-readable media ofclaim 13, wherein setting the 3D position of the lighting unit comprisesregressing the reflected 3D rays to the 3D position of the lighting unitby applying least squares regression to the reflected 3D rays.
 15. Theone or more non-transitory computer-readable media of claim 12, whereindetermining the 3D orientation of the lighting unit comprises:determining 3D positions for a set of corner vertices of the lightingunit; determining a plane in 3D space that intersects the set of cornervertices of the lighting unit; computing a normal vector of the plane;and selecting a direction of the normal vector that is towards thecenter of the mirror sphere.
 16. The one or more non-transitorycomputer-readable media of claim 12, wherein determining the intensityassociated with the point light source comprises: projecting a first 3Dray from a first pixel in a first captured image included in theplurality of captured images towards the mirror sphere; reflecting thefirst 3D ray off the mirror sphere to generate a first reflected 3D ray;determining that the first reflected 3D ray intersects a first pointlight source in the plurality of point light sources; and setting theintensity associated with the first point light source to an intensitycontribution of the first pixel.
 17. The one or more non-transitorycomputer-readable media of claim 16, wherein determining the intensityassociated with the point light source further comprises adjusting theintensity contribution of the first pixel based on an attenuation oflight along the first 3D ray as light from the first 3D ray travels fromthe first point light source to the mirror sphere.
 18. The one or morenon-transitory computer-readable media of claim 17, wherein theattenuation of the light along the first 3D ray is based on aninverse-square-distance law.
 19. The one or more non-transitorycomputer-readable media of claim 16, wherein determining the intensityassociated with the point light source further comprises: projecting asecond 3D ray from a second pixel in the first captured image towardsthe mirror sphere; reflecting the second 3D ray off the mirror sphere togenerate a second reflected 3D ray; determining that the secondreflected 3D ray intersects the first point light source; accumulatingthe intensity associated with the first point light source with anintensity contribution of the second pixel to generate an accumulatedintensity; and setting the intensity associated with the first pointlight source to the accumulated intensity.
 20. A system, comprising: oneor more memories storing instructions; and one or more processorscoupled to the one or more memories and, when executing theinstructions: determine at least one of a three-dimensional (3D)position or a 3D orientation of a lighting unit based on a plurality ofcaptured images of a mirror sphere; for each point light source in aplurality of point light sources included in the lighting unit,determine an intensity associated with the point light source; captureappearance data of an object, where the object is illuminated by thelighting unit; and render an image of the object based on the appearancedata and the intensities associated with each point light source in theplurality of point light sources.